For the 2023-2024 influenza season, CDC called for national and jurisdiction categorical probability predictions for the direction and magnitude of changes in hospitalization rates per 100k population. The collection period began October 11, 2023 and will run through May 1, 2024.

This report is supported by work from colleagues in the Reich Lab at UMass-Amherst, Jessica Davis at Northeastern University, and Gursharn Kaur and Srini Venkatramanan at University of Virginia.

Categorical forecasts are evaluated using the Brier Score, Brier Skill Score, and Ranked Probability Score. All scores are defined below. Models presented include the FluSight-ensemble - an ensemble of submitted categorical forecasts and FluSight-equal_cat which is used as a baseline comparison and is comprised of equal probabilities (0.2) for each category in each jurisdiction each week.

Higher skill scores are better

Forecasts after 11/03/2023 are included.

Brier Score and Ranked Probablity Score

Definitions

Let \(\{-2,-1,0,1,2\}\) be the numeric representation of the categories {“large_decrease”, “decrease”, “stable”, “increase”, “large_increase”}. Then for a model \(m\), location \(loc\), and horizon \(h\)

Ranked Probability Scores is given by: \[ RPS(m, loc, h) = \frac{1}{4}\sum_{i=-2}^2 \left( \hat{F}(i)- F(i)\right)^2\] where \(\hat{F}\) and \(F\) denotes the forecast CDF and observed CDF on the discrete state space \(\{-2,-1,0,1,2\}\).

Brier score is given by: \[ BS(m, loc, h) = \sum_{i=-2}^2 \left(p(i)- p(i)\right)^2\] where \(p(i)\) and \(o(i)\) denotes the forecast probability and observed probability for category \(i\).

Average RPS (for a model \(m\) and location \(loc\)) is given by: \[ RPS(m, loc) = \frac{1}{4} \sum_{h=0}^3 RPS(m, loc, h)\]

Similarly, average Brier score (for a model \(m\) and location \(loc\)) is given by: \[ BS(m, loc) = \frac{1}{4} \sum_{h=0}^3 BS(m, loc, h)\]

Skill scores are taken as 1 minus the pairwise geometric mean of a particular model (compared to all other submitted models) over the pairwise geometric mean of the equal probability model.

Example: RPS vs BS

Evaluation Metrics

Recent Accuracy

Tables

RPSS
BSS

Boxplots

RPSS

RPSS for model across locations

BSS

BSS for model across locations

Historical Accuracy

Tables

RPSS
BSS

Boxplots

RPSS

RPSS for model across locations

BSS

BSS for model across locations

Metrics by model

Brier

Horizon 0

Horizon 1

Horizon 2

Horizon 3

RPS

Horizon 0

Horizon 1

Horizon 2

Horizon 3

Evaluation by location

Using all data

Ranked Probability Skill Score

Brier Skill Score

Forecast trend for reference date April 20, 2024

Probability of any increase

Probability of any decrease

Evaluation Periods